THE ASYMPTOTICS OF THE RAY-SINGER ANALYTIC TORSION OF 

HYPERBOLIC 3-MANIFOLDS 



WERNER MULLER 

' Abstract. In this paper we consider the analytic torsion of a closed hyperbolic 3- 

manifold associated with the m-th symmetric power of the standard representation of 
M ' SL(2,C) and we study its asymptotic behavior if m tends to infinity. The leading coeffi- 

cient of the asymptotic formula is given by the volume of the hyperbolic 3-manifold. It 
follows that the Reidemeister torsion associated with the symmetric powers determines 
\^ I the volume of a closed hyperbolic 3-manifold. 

(N 

S7 

C/^ ■ 1. Introduction 

I Let X be a closed, oriented hyperbolic 3-manifold. Then there exists a discrete, torsion 

free, co-compact subgroup T C SL(2, C) such that X = T\B.^, where = SL(2, C)/ SU(2) 
is the 3-dimensional hyperbolic space. Let p be a finite-dimensional complex representation 
of F and let Ep ^ X he the associated flat vector bundle. Choose a Hermitian fiber metric 
J> I h on Ep. Let Tx{p] g, h) denote the Ray-Singer analytic torsion of the de Rham complex 
^ ■ of i?p- valued differential forms [[RSj , where g denotes the hyperbolic metric. If p is acyclic, 
then Tx{p;g,h) is metric independent [[Mul| , Corollary 2.7]. In this case we denote it by 

^ : Txip). 

2 . For m G N let = Sym™ be the m-th symmetric power of the standard representation 

O ! of SL(2,C) on and denote by Er^ the fiat vector bundle associated to r^lr- It is 
well known that H*{X, Er^) = 0. This follows, for example, from |PW| , Chapt. VII, 
Theorem 6.7]. Hence the restriction of to F is an acyclic representation of F. Denote 
^ . by Txijm) the analytic torsion with respect to Tm\v- The purpose of this paper is to study 
^ ! the asymptotic behavior of Tx{Tm) as m — > oo. Our main result is the following theorem. 

Theorem 1.1. Let X be a closed, oriented hyperbolic 3-manifold. Then we have 

(1.1) - log Tx(r„) = ^-^^m^ + 0(m) 

as m — 7- oo. 



We note that there is an analogous result in the holomorphic setting. In ||BV|] Bismut 
and Vasserot studied the asymptotic behavior of the holomorphic Ray-Singer torsion for 
symmetric powers of a positive vector bundle. 
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Let Tx{Tm) denote the Reidemeister torsion of X with respect to r^lr (see ||Mul|| ). Then 
by ||Mul| , Theorem 1] we have Tx{Tm) = Tx{Tm)- Thus we obtain the following corollary. 

Corollary 1.2. Let X be a closed, oriented hyperbolic 3-manifold. Then we have 

(1.2) - log rx(r„) = ^^^^m^ + 0(m) 

as m ^ CO. 

This result has applications to the cohomology of arithmetic hyperbolic 3-manifolds. We 
will discuss this elsewhere. As an immediate corollary we get 

Corollary 1.3. Let X be a closed, oriented hyperbolic 3-manifold. Then vol(X) is deter- 
mined by the set of Reidemeister torsions {Tx{Tm)'- fn ^ N}. 

Some remarks are in order. The Reidemeister torsion of a compact 3-manifold is known 
to be a topological invariant [ph|] . Therefore, it follows from Corollary p..3| that the volume 



of a compact, oriented hyperbolic 3-manifold is a topological invariant. This is also a well 



known consequence of the Mostow-Prasad rigidity theorem ||Mo| , 

There are only finitely many closed, oriented hyperbolic 3-manifolds with the same volume 
|P?1| , Theorem 3.6]. Therefore we get 

Corollary 1.4. A compact, oriented hyperbolic 3-manifold X is determined up to finitely 
many possibilities by the set {rx{Tm) : m e N} of Reidemeister torsion invariants. 

It is known that the number of closed hyperbolic manifolds with a given volume can 
be arbitrarily large. Therefore the proof of the corollary does not give a uniform bound 
on the number of closed hyperbolic manifolds with the same set of Reidemeister torsion 
invariants. 

Our approach to prove Theorem [L.l| is based on the expression of Tx{Tm) in terms of the 
twisted Ruelle zeta function attached to Tm- Recall that for a finite-dimensional complex 
representation p of F the twisted Ruelle zeta function Rp{s) is defined for Re(s) ^ as 
the infinite product 

(1.3) Rp{s)= ndet(ld-p(7)e-^^W), 

prime 

where [7] runs over the nontrivial primitive conjugacy classes of F and £(7) denotes the 
length of the unique closed geodesic associated to [7]. It follows from |[I'r2|| that Rp{s) 



admits a meromorphic extension to the whole complex plane. If p is unitary and acyclic 
then Rp{s) is regular at s = and its value at zero satisfies 

\RM\ = Tx{pf 

(see |F^). For an arbitrary unitary representation (which is not necessarily acyclic), the 
coefficient of the leading term of the Laurent expansion of Rp{s) at s = is given by the ana- 
lytic torsion. The corresponding result holds for any compact, oriented hyperbolic manifold 
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of dimension n . In his thesis |[Br|| U. Brocker has estabhshed a similar resuh for repre- 



sentations of the fundamental group that are restrictions of finite-dimensional irreducible 
representations of the isometry group SOo(n, 1) of the hyperbolic n-space. Unfortunately, 
his method is based on elaborate computations which are difficult to verify. This problem 
has been rectified by Wotzke in his thesis ||Wo|| . He gave a different proof which replaces 



Brocker's explicite computations by the real version of Kostant's Bott-Borel-Weil theorem 

To state the result for n = 3 we need to introduce some notation. Let r be a finite- 
dimensional, irreducible representation of SL(2, C), which we regard as real Lie group. Let 
6 be the Cartan involution of SL(2, C) with respect to SU{2). Put Tg = t o 9. Denote by 
Rt{s) the twisted Ruelle zeta function for the restriction of r to F. Let E^- X he the 
flat vector bundle associated to r|r. The flat bundle Er can be equipped with a canonical 
Hermitian fiber metric ||MM|| . Let Ap(r) be the corresponding Laplacian on i?^-valued 



p-forms and denote by Tx{t) the Ray-Singer analytic torsion associated to r|r. Then the 
main result of ||Wo|| for n = 3 is the following theorem. 



Theorem 1.5. Let t be a finite- dimensional, irreducible representation of the real Lie 
group SL(2,C). Then we have 

1) If T0 ^ r, then Rt{s) is regular at s = and 

\RAO)\=Tx{tY. 

2) Let Tg = T. If T ^ 1, then the order h{T) of Rr{s) at s = is given by 

3 

(L4) /i(r) =2^(-l)PdimkerAp(r). 

p=i 

and if T = 1, the order equals 4 — 2 dim iJ^ (X, M) . The leading term of the Laurent 
expansion of Rr{s) at s = is given by 

TxirYs'^^^l 



The case of the trivial representation is covered by f^. In this case the order of R{s) at 
s = differs from (11 



It follows from Theorem |1.5| that in order to prove Theorem |1.1| it suffices to analyze 
the asymptotic behavior of Rr^{0) as m — )■ oo. For this purpose we consider another type 
of twisted Ruelle zeta functions. Let A be the standard split torus of SL(2, C) and let 



M be the centralizer of A in SU(2) (see (2.1) for the explicit description). For a E M 



let R{s,a) be the Ruelle zeta function defined by (|3.6|). Using the decomposition of t.^ 
under the Cartan subgroup MA, it follows that Rt^{s) is the product of the twisted 
Ruelle zeta functions with shifted argument R{s — (m/2 — k),am-2k), k = 0, ...,m. This 
reduces the study of the asymptotic behavior of Tx{j2m) (resp. Tx{j2m+i)) as m — )• oo to 
the study of the behavior of \R{k,a2k)\ and \R{—k,a2k)\, (resp. \R{k + l/2,cr2fc+i)| and 
\R{—k — 1/2, cr_(2fc+i)|). A; > 2, as /c — !■ oo. To analyze the behavior of \R{k,a2k)\ (resp. 
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\R{k + 1/2, cr2fc+i)|) as A; — 7- oo we simply use the infinite product defining it. To deal with 
the remaining cases we use the functional equation which implies 

=exp(-47r-ivol(X)Re(s)) 

This is exactly how the volume of X appears in the asymptotic formula (IT] 



For the sake of completeness we include a proof of Theorem p..5| which is based on results 
of ||B0|| . The starting point of the method is the observation that the flat bundle Er ^ X 



is isomorphic to the locally homogeneous vector bundle defined by the restriction of r to 
the maximal cocompact subgroup SU(2) of SL(2, C) (see ||MM| , Propostion 3.1]). Using 
this isomorphism the bundle Er can be equipped with a canonical Hermitian fiber metric 



induced from an invariant metric on the corresponding homogeneous vector bundle ||MM 
Lemma 3.1]. We define the Laplacian Ap(r) on E'^-valued p-forms with respect to this 
metric on Er- Then, up to a constant, Ap(r) equals —R{fl), where R{Q) denotes the 
action of the Casimir operator on sections of the locally homogeneous bundle. This is the 
key fact which allows us to apply the Selberg trace formula to the heat kernel. 

The paper is organized as follows. In section |] we summerize some basic facts about 
hyperbolic 3-manifolds and analytic torsion. In section ^ we consider twisted Ruelle and 
Selberg zeta functions and establish some of their basic properties. In section]^ we introduce 
certain auxiliary elliptic operators which are needed to derive the determinant formula and 
to prove the functional equation for the Selberg zeta function. In the next section ^ we 
establish the functional equation for the Selberg and Ruelle zeta functions. In section 



we use the determinant formula of [[BU|] to express the twisted Ruelle zeta function as a 



ratio of products of regularized determinants of the elliptic operators introduced in section 
^. The determinant formula is one of main tools to study the leading term of the Laurent 
expansion of Rt{s) at s = 0. In section |^ we give a proof of Theorem |1.5| . In the final 



section ^ we proof Theorem |L1 
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2. Preliminaries 



2.1. Let G = SL(2, C) and K = SU(2). Then is a maximal compact subgroup of G. We 
regard G as real Lie group and we recall that G is isomorphic to SpinQ(3, 1) P^] . Under 
this isomorphism, SU(2) is mapped to Spin(3). Thus G acts on the hyperbolic 3-space 
and = G/K. Let P C G be a cocompact, torsion free discrete subgroup. Then 
X = P\EI^ is a compact, oriented hyperbolic 3-manifold and any such manifold is of this 
form. 

Let G = NAK be the standard Iwasawa decomposition of G and let M be the centralizer 
of A in K. Then 

(2.1) ^={(o A°i)^^^^^}' ^={("0' ^^e[0,27r]|. 
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We use the natural normalization of the Haar measures for A,N,K and G as in |[Kn| , pp. 
387-388]. In particular, we choose on K the Haar measure dk of total mass 1. 

Let g, i, a, m and n denote the Lie algebras of G, K, A, M and N, respectively. Let 

(2.2) 5 = t®p 

be the Cartan decomposition of (g, a). Then a is a maximal abelian subspace of p. Let a 
be the unique positive root of (g, a). Let H E a he such that a{H) = 1. Let a+ C a be 
the positive Weyl chamber and let A'^ = exp(a"^). Let W := W{g, a) denote the restricted 
Weyl group. 

Put f) = m © o. Then f) is a Cartan subalgebra of g. We identify f) with M^. Then 
the Weyl group Wq of (gc, f)c) acts on by sign changes. So Wg has order 4. In a 
compatible ordering on t)^ the only positive roots of the pair (gc, f)c) are ai and a2 where 
ai{H) = a2{H) = a{H) = 1 and aiiiH) = —a2{iH) = i. Let pc = + 012). 

Let B be the Killing form of g. Define a symmetric bilinear form on g by 

(2.3) (Fi, Y2) = \b{Y^, Y2), Fi, ^2 G g. 

Then (-, ■) is positive definite on p, negative definite on t and we have (C, p) = 0. The 
normalization is such that the restriction of (■, ■) to p = T^xiG/K) induces the G-invariant 
Riemannian metric on M.^ = G/K which has constant curvature —1. 

Let {Zi} be a basis of g and let {Z^} be the basis of g which is determined by {Zi, Z^) = 
Sij. Then the Casimir element fl e Z{qc) is given by 

(2.4) ^] = ^ZiZ^ 

i 

Let R{Q) be the differential operator induced by Q on C°°{M.^). Then by Kuga's lemma 
we have 

R{n) = -A, 

where A is the hyperbolic Laplace operator on functions. 

2.2. Let r C G be a discrete, torsion free, cocompact subgroup. Then X = r\H^ is a 
closed hyperbolic manifold. Given 7 G F, we denote by [7] the F-conjugacy class of 7. The 
set of all conjugacy classes of F will be denoted by C(F). Let 7 7^ L Then there exist 
g E G, rrij G M, and G A'^ such that 

(2.5) g-fg'^ = m^a^. 



By ||Wal| , Lemma 6.6], depends only on 7 and is determined up to conjugacy in M. 



By definition there exists £(7) > such that 

(2.6) = exp (£(7)//) . 

Then £(7) is the length of the unique closed geodesic in X that corresponds to the conjugacy 
class {7}. An element 7 G F — {e} is called primitive, if it can not be written as 7 = 7o 
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for some 70 € F and A; > 1. For every 7 G F — {e} there exist a unique primitive element 

7o G F and nr(7) G N such that 7 = 'j^^^'^K 

2.3. Denote by M the set of unitary characters of M. Then M = Z and the character ak 
that corresponds to G Z is given by 

(2.7) -'((C e-))=^"" 

The finite-dimensional irreducible representations of G, regarded as real Lie group, are 
parametrized by pairs of nonnegative integers |[Kn| , p. 32]. For m G No let 



Tm = Sym'": G ^ GL(S'"(C2)) 

be the m-th symmetric power of the standard representation of G = SL(2, C) on C^. 
Denote by the complex conjugate representation. Then 

(2.8) Tm,n Tm ® Tn 

is the irreducible representation with highest weight {m,n). The restrictions of to MA 
decomposes as follows: 



(2.9) r^l^^^ = ^crm_2fc ® e^2 



m 



k=0 



2.4. Let P = MAN be the standard parabolic subgroup of G. We identify C with by 
z — )■ zH. For G Z and A G C let 7r„ ^ be the induced representation 

(2.10) 7r„,A = Ind^(cXn®e*^®l). 

Note that this is the parametrization of the principal series used in | [Kn| , Chapt. XI, §2]. 
The representation Hn,x acts in the Hilbert space Tin^x whose subspace of C°°-vectors is 
given by 
(2.11) 

= {/ e C°^(G,K): figman) = e-(^^+i)(i°^'^)a(m)-V(^7), 9 G G, man G P} . 

If A G M, then vr^ ^ is unitary. This family of representations is the unitary principal series. 
All 7rn,x, G Z, A G M \ {0}, are irreducible. They have an explicit realization [|Kn| , Chapt. 
II, §4]. The Casimir eigenvalue 7r„^A(^) is given by 

71 ^ 

(2.12) 7rr,,xm = -A2 + - - 1. 

This follows from [ [Kn| , Theorem 8.22]. It can be also verified using the explicit realization 
of 7r„ A- In the latter case one has to take into account that the identification of Oc with C 
is differnt from ours. 

The nonunitarily induced representations 

(2.13) TT^ = Ind^(l®e^®l), < x < 2, 
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are unitarizable. This is the complementary series. The Casimir eigenvalue is given by 
(2.14) <(n) =x^-l. 

This also follows from |[Kn| , Theorem 8.22] or can be verified using the explicit realization 
of TT^ (see ||Kn| , Chapt. 11, §4]). Denote by 0„^a = ti^Tr^ a the character of 7r„ a- 



2.5. Let r: G — )■ GL(V^) be an irreducible finite-dimensional representation of G. Let E^- 
be the fiat vector bundle associated to the restriction r|r of r to F. By ||MM| , Proposition 



3.1] Er is canonically isomorphic to the locally homogeneous vector bundle associated to 
t\k, i.e., 

(2.15) Er = {T\G X Vr)/K, 

where K acts on r\G xVrhj [Tg, v) ■ k = {Tgk, T{k)^^v). So we may regard Er as locally 
homogeneous vector bundle equipped with a fiat connection which, of course, is different 
from the canonical invariant connection on the homogeneous bundle. 

The vector bundle Er can be equipped with a canonical fiber metric. By ||MM , Lemma 
3.1] there exists an inner product (■, ■) on Vr which satisfies 

(a) {t{Y)u,v) = -{u,t{Y)v) for all Y et,u,v e K; 

^^■"^^^ (b) {t{Y)u, v) = {u, t(Y)v) for all Y ep,u,ve 

Such an inner product is called admissible. It is unique up to scaling. By (a) the inner 



product is invariant under t(K) and therefore, it defines via ( |2.15| ) a Hermitian fiber metric 
in Er- Denote by Ap(r) the Laplacian on ii^^- valued p-forms with respect to an admissible 
metric on Er- 

2.6. Let P be an elliptic differential operator acting on C°°-sections of a smooth Hermitian 
vector bundle E over a compact Riemannian manifold X. The metrics g on X and h on 
E induce an inner product in G°°{X, E). Suppose that with respect to this inner product 
the operator P is symmetric and nonnegative. Then the zeta function (^(s;P), s G C, of 
P is defined as 

C(s;P)= rn{\)\-\ 

AeSpoc(P)\{0} 

where m(A) denotes the multiplicity of the eigenvalue A. The series converges absolutely 
and uniformly on compact subsets of Re(s) > dim(X)/ord(P). Moreover C{s] P) admits a 
meromorphic extension to s G C which is holomorphic at s = (see |5Ii, Chapt. 11]). Then 
the regularized determinant det P of P is defined as 

(2.17) detP = exp(^-^C(^;^)L=o)- 

Assume that P is symmetric and bounded from below. Let A G M be such that P + A > 0. 
Then det(P + A) is defined by ( p. 17] ). Voros [ |Vo|] has shown that the function A det(P- 



A), defined for A ^ 0, extends to an entire function det(P + s) of s G C with zeros —Xj 
where Xj G Spec(P). 
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2.7. Finally we recall the definition of the Ray-Singer analytic torsion |^^. Let x be 
a finite-dimensional representation of t^i{X) and let i?^ — > X be the associated flat vec- 
tor bundle over X. Pick a Hermitian fiber metric h in and let Ap(x) : A^(X, E-^ — )■ 
A^(X, E-^ be the Laplacian on the space of i?-^- valued p- forms. Then Ap(x) is a nonnega- 
tive, second order elliptic differential operator. So it has a well defined regularized deter- 
minant, defined by ( ^.17| ). Then the analytic torsion is defined as the following weighted 
product of regularized determinants 

3 

(2.18) Tx(x; 9,h) = \{ (det ^ix))^-'^"^'"'^ . 

p=i 

By definition Tx{Xj 9i h) depends on g and h. However, if dimX is odd and x is acyclic, 
i.e., H*{X,E^) = 0, then Tx{Xj9yh) is independent of g and h ||Mul| , Corollary 2.7]. In 
this case we denote it simply by Tx(x). 

In this paper we consider the special case where X = r\M.^ is a closed hyperbolic 3- 
manifold and x is the restriction of a representation t: G ^ GL{Vr) to F. Then, as 
explained above, the flat bundle E^. carries an admissible metric. We denote the analytic 
torsion attached to r|r with respect to this metric by Tx{t). 

3. Twisted Ruelle and Selberg zeta functions 

In this section we consider various kinds of twisted geometric zeta functions which are 
needed for the proof of our main result. We will use the notation introduced in section ^ 
First we recall the following estimation of the growth of the length spectrum. For i? > 
we have 

(3.1) #{[7]GC(F):%)<i?}«e^« 

m (1-31)]. 

If T is an endomorphism of a finite-dimensional vector space, denote by S^T the /c-th 
symmetric power of T. Let n = be the negative root space. Then for a G M and s G C 
with Re(s) > 2 the twisted Selberg zeta function is defined by 

oo 

(3.2) Z{s, f^) = n n " (^(^-^^ ® {M{m^a^)^)) e-(^+i)^W) , 

[7]^efc=0 
prime 

where [7] runs over the non-trivial primitive conjugacy classes in F. By | [BU| , (3.6)] we have 



(3.3) \ogZ{s,a) = -Y, 



det(Id — Ad(m^a^)n)/i(7) 



e 



It follows from ( p.l|) that the series converges absolutely and uniformly in the half-plane 
Re(s) > 2. Therefore the infinite product converges absolutely and uniformly in the half- 
plane Re(s) > 2. Furthermore by [ [B0| , Theorem 3.15] it has a meromorphic extension to 
the entire complex plane and satisfies a functional equation [BD, Theorem 3.18]. To state 



9 



the functional equation we need some notation. Let w G Wa be the non-trivial element. 
Then w acts on M by 

{wa)(m) = a(m^^mmw), m G M, a G M, 

where rriw is a representative of w in the normalizer of a in i^'. Thus wak = cf-ki A; G Z. 
For each a G M there is an associated Dirac operator Dx{cr) acting in a Clifford bundle 
— 7- X P0| , p. 29]. Let ?7(Z)x(c")) denote the eta invariant of Dx{(j)- Let Pa be the 



Plancherel polynomial with respect to a. If the Haar measures are normalized as in ||Kn| , 
pp. 387-388] and ac is identified with C by 2; G C h- )■ zH G ac, then by |[Kn| , Theorem 11.8] 
(up to a minor correction) it is given by 

(3.4) ^-(^) = 4^(t-"')' 

where cxfc G M is the character defined by ( p.?] ). We note that our definition of Pa{z) differs 
from the definition of Prr(z) in ||BO, 



P- 



Then the functional equation satisfied by Z{s, a) is the following equality 
(3.5) Z(s,a) = e^"^(^^("))exp |-47rvol(X)^ P^ir) dr"^ Z{-s,wa) 



B0[ Theorem 3.18]. Our formula differs from the formula in [P0| , Theorem 3.18]. This 
is due to the the different definition of Pa{z). Since the functional equation plays an 
important role in this paper, we will give a separate proof for the functional equation of 
the symmetrized Selberg zeta function in section |^. 

A related dynamical zeta function is the twisted Ruelle zeta function R{s,a) which is 
defined by 

(3.6) R{s, a)=Y[{l- a{m^)e-''^^^) , 

prime 

where, as above, [7] runs over the non-trivial primitive conjugacy classes in F. Note that 
R{s, (To) equals the usual Ruelle zeta function 

(3.7) R{s) = n ~ ^~''^''^) • 

prime 



Ruelle zeta functions of this type have been studied by Fried, and Bunke and Olbrich |[B0 
For any o" G M let ri{Dx{(T)) be the eta invariant occurring in the functional equation 
( p. 51) . The two zeta functions are closely related. Namely the Ruelle zeta function can be 
expressed in terms Selberg zeta functions as follows. 

Lemma 3.1. For every a & M we have 

Z{s + l,a)Z{s-l,a) 



(3.8) R{s, 



a] 



Z{s, a 0-2)^(5, a ® a-2) ' 
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Proof. By ^ (3.4)] we have 

(3.9) logi?(.,a) = -5^^e~^^W. 

Using ( p. 3D we get 

logZ(s + 1, a) + log Z{s - 1, a) - log Z{s, cr ® (T2) - log Z{s, a ® (j_2) 
^ _ aK)(l - a2K)e-^(^) - a-2K)e-^(^) + e-^^(^)) ^_,,(^) 

(3.10) det(Id-Ad(m^a^)i^)nr(7) 

Together with ( |3.9| ) the lemma follows. □ 
Put 

9x{(r) := 2r]{Dx{cr)) - r]{Dx{(r ® ^2)) - viDx{(r ® a-2)), a e M. 
We summarize the main properties of R{s, a) by the following proposition. 

Proposition 3.2. For each a & M we have 

1) The infinite product ^3. (\ ) is absolutely convergent in the half-plane Re(s) > 2. 

2) R{s,a) admits a meromorphic extension to whole complex plane. 

3) R{s, a) satisfies the following functional equation. 



(3.11) R{s, a) = e^"^^(") exp {Att"^ vol(r\tf )s) R{-s, 



wa]. 



Proof. 1) follows from the estimation ( p.l| ). The meromorphic extension is established in 
pOI , Chap. 4] and the functional equation is proved in ||B0| , Theorem 4.5]. It follows from 
Lemma ^]l| and the functional equation of the Selberg zeta function. Namely using ( |3.8| ) 
and ( |3.5|) we get 



R{s,(t) 
R{—s, wa) 



^inexia) ( yoK^x) I ^ P,^ (r) dr + P,^ (r 



Jo 

It follows from ( |3.4| ) by a simple computation that 

"S+l ps — l r-s j-s ^ 

P^Ar) dr + / P.,(r)g - / P,,^,(r) dr - / P.,_,(r) dr = - — 



^0 ^0 ^0 



which implies 3). □ 
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Now let r: G — t- GL(l^) be a representation in a finite-dimensional complex vector space 
V. We fix a norm || ■ || inV. The restriction t\ma of r to MA decomposes into characters: 



(3.12) r|MA = 0fffc®e^ 



where J C Z is finite and z/^ G |Z. Let c = max{|z/fc| : k E I}. Given g E G, we denote by 
a{g) G A'^ the ^"'"-component of g with respect to the Cartan decomposition G = KA^K. 
It follows from ( |3.12| ) that there exists Ci > such that 

Ik(^) ||< Cie'^^C^s^^^", geG. 
This implies that there exists C2 > such that 

II r(7) ||< Ce'^^'(^), 7Gr\{l}. 
Therefore, the infinite product 

(3.13) R^{s) = W det (l-r(7)e-^^(^)) 

prime 

is absolutely convergent in the half-plane Re(s) > C2 + 2. By ( p.l2|) we have 

det (l-r(7)e-"^(^)) = det (l -r(m^a^)e-^^(^)) = JJdet (l - afc(m^)e-(^-'^'=)^(^)) . 

Taking the product of both sides over all non-trivial primitive conjugacy classes, we get 

(3.14) R,{s) = \{R{s-Vk,<yu), Re(s)>C2 + 2. 

fee/ 

The right hand side is a meromorphic function on C. This implies that Rt{s) admits a 
meromorphic continuation to C. 



Using (|3.8| ), it follows that Rt{s) can also be expressed in terms of twisted Selberg zeta 
functions. This formula can be simplified using Kostant's Bott-Borel-Weil theorem ||Ko 
which we recall next. Let 

(3.15) fip-. MA^ GLiA^nc), p = 0,1,2, 

be the p-th exterior power of the adjoint representation of MA on xiq- It decomposes into 
characters as follows 



(3.16) /io = do, yUi = ((72 ® e°) © (a-_2 ® e"), /i2 = o-q ® e 



2a 



Denote by fip the contragredient representation of the representation ( |3.15| ). Given (m, n) G 
Z X Z, we define a character X{m,n) '■ MA — )■ by 



(3.17) X{m,n) = (^m-n ® 6 2 
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Lemma 3.3. Let r be an irreducible representation of G with highest weight E NqxNq. 
We have the following identity of characters of MA. 

2 

(3.18) ^(-irtr/i,-trr= (-ir^'^^X.CA.+poWo, 
where i{w) denotes the length of w. 

Proof. Let L = MA. For a finite-dimensional L- module W denote by ch.L{W) the element 
in the character ring R{L). Let Vr be an irreducible G-module with highest weight At-. By 
the analog of a result of Kostant [|Ko| , Theorem 5.14], for real Lie algebras [p3W| , Theorem 
III.3.1], i|, we have 

2 

(3.19) 5^(-l)^ch^(ff^(n,K)) = Yl (-l)'^"'^XMA.+Po)-Pc, 

where HP{n, Vr) denotes the Lie algebra cohomology. By the Poincare principle [[Ko| , (7.2.3)] 
we have 

2 2 

(3.20) 5^(-l)^'chi(Afn*®K) = 5Z(-l)^chi(ifP(n,K)). 

p=0 p=0 

Here L acts on n* via the contragredient representation of the adjoint representation. 
Combining ( |3.19| ) and ( p. 201) , the lemma follows. In fact, in the present case the lemma 
could also be proved by an elementary computation, using the parametrization ( |2.8| ). □ 

We are now ready to prove the formula which expresses Rt{s) as a fraction of twisted 
Selberg zeta functions. For w G Wq write 

(3.21) XHAr+PG)-PG = ^r,u, ® e(^^---i)", 
where ar^w ^ M and Xt-m ^ IR. 

Proposition 3.4. Let r be an irreducible finite- dimensional representation of G. Then we 
have 

(3.22) Rr{s)= n ^(3-A..^,a.,^)(^i)''""^\ 

w^Wg 

Proof. Recall that for an endomorphism of a finite-dimensional vector space we have 

oo 

(3.23) det(Id -W) = tT{A''W). 

Let m E M and a E A. Note that fipima) = A'^ Ad{ma)n. Hence if we apply (|3.23|) to jlp 
we get 

2 

^^(—1)^ tr jlp{ma) = det (Id — Ad(ma)n) . 

p=0 
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Using (^) and (lOTI) , we get 

.4^0 det(Id-Ad(ma)^) 



Next we have 



logi?,(s) = J2 ^^log (ld-r(7)e-'^^(^)) 

tr (r(7)e-^^W)'' 



prime 



(3.25) = - 5Z 5Z 



k 

k=l 

prime 



Now let 7 G r \ {e}, 7 ~ m^a^. Then loga^ = i{'~f)H. Inserting (p.24|) on the right hand 



side of (|3.25|) , we get 



e{w)+i o'T,w{m^) ^_(s_A,,„+i)£(-y) 

det (Id — Ad(m^a^)n-) n^(7) 



iogi?.(.) = J2 i-iY^""^^' E - ' 



By [ P0| , (3.6)], the right hand side equals 

^ (-l)^M+MogZ(s-A.,^,a.,J, 

which proves the proposition. □ 

We also need to consider symmetrized Ruelle and Selberg zeta functions. Recall that the 
nontrivial element wa G Wa acts on M by WA^k = cr-k- Let a E M \ {o"o}. Put 

(3.26) S{s^a) := Z{s,a)Z{s,WA(T)- 

This is the symmetrized Selberg zeta function. Let 6': G — ?• G be the Cartan involution. 
Put 

(3.27) T0=TOe. 

Note that Tp = Sym^ satisfies Tp o $ = Tp, and TpO 6 = Tp. Thus 6 acts on the highest 
weights by 

(3.28) 6{m,n) = {n,m), (m,n)G No xNq. 



By ( |2.8| ) it follows that an irreducible finite-dimensional representation r of G satisfies 
T0 = T, if and only if r = Tm,m for some m G Nq. 



14 WERNER MULLER 

Proposition 3.5. Let r be an irreducible finite- dimensional representation of G. Then we 
have 

(3.29) Rr{s)Rr,{s)= H 5(s-A,,^,cr.,^)(-i)''"''^\ re ^ r, 

w£Wc 

and 

(3.30) Rr^^{s) = Z{s- {m + l),ao)Z{s + m + l,ao)S{s,a2m+2)~^, m G Nq. 
Proof. Put 

S(r) = {w{Ar + Pg) - Pg- w e Wg}- 
Let r = Tm^n- Then we have 

S(r) = {(m, n), (-(m + 2),n), [m, -{n + 2)), (-(m + 2), -(n + 2))}. 
By ( |3T7| ) and ( p:21| ), it follows that 

{icrr,w, K,w) ■■ w e Wg} = { (cTm-n, {m + n)/2 + 1) , (a_(„+„+2), {n - m)/2) , 

{am+n+2, {m - n)/2) , {an-m, -{m + n)/2 - 1)}. 
Assume that m ^ n. Using that (rm,n)e = Tn,m and ( |3.31| ), it follows that 
{{(Tr,w, K,w) ■ w e Wg} U {{arg,w, Kg,w) ■ w G Wg} 

= {{(^T,w, K,w), {wa<^t,w, K,w) ■ w G Wg} . 

By ( |3.22| ) the first equality follows. Now assume that tq = r . By ( p.28[ ) there exists 
m G No such that r = Tm,m- In this case we get 

(3.32) {icrr,u>,K,w)- w G Wg} = {(ao,m + 1), (o-_2(m+i), 0), (a2(m+i), 0), (ctq, -(m + 1)} . 
Using again and d^M) : we get (lOoD . □ 

4. Bochner-Laplace operators 

In this section we study certain auxiliary elliptic operators which are needed to derive 
the determinant formula and the functional equation for the Selberg zeta function. These 
operators were first introduced by Bunke and Olbrich ||BU|| . 

Let wa G Wa be the nontrivial element. It acts on dk G M by WAO'k = (^-k- Thus, 
if A; 7^ 0, then 0"^ is not VF^-invariant. For / G Nq let vi & K denote the irreducible 
representation of i^' = SU(2) of highest weight /. Then we have 

I 

(4.1) vi\m = ^ <yi-2k- 

k=0 

Let R{K) and R{M) denote the representation rings of K and M, respectively. The 
inclusion i: M ^ K induces the restriction map i* : R{K) — )■ R{M). From ( |4.1| ) we get 

i*{ui - =(yi + a^i, / G N, / > 2; 

r(z/i) = (Ji + (7_i, i*{uo) = cro. 
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It follows from ([4 .21 ) that for every a E M there exists a unique £ R{K) such that 

(4.3) i*{i^) = a + WA<y. 
Then we have 

(4.4) i^ = ^m,{a)u. 

with rriyia^ E {0, ±1} for o" 7^ (Tq and m^^{(TQ) = 0, if / 7^ 0, and m^g{ao) = 2. 

Given u E K, let E^, denote the associated homogeneous vector bundle over G/K and 
El, = T\Ei, the corresponding locally homogeneous bundle over X. For a E M and u E K 
let my{(T) be defined by (^. Put 

(4.5) E{a)= E,. 

V 

This bundle has a canonical grading 

(4.6) E{cx) = E+{a)®E-{cx) 
defined by the sign of m^{a). 



Let be the elliptic G-invariant differential operator on C°°{G/K, E^) = (C°°(G)(g)V; 
which is induced by —Q, where Q E 2(0c) is the Casimir element. Let 

A, = (V")*V" 

be the connection Laplacian associated to the canonical invariant connection of E,y. By 
||Mi4 Proposition 1.1] we have 



(4.7) = A, - u{n 



K 



where VIk E Z{l£) is the Casimir element of K. Being G-invariant, Ai, descends to an 
elliptic operator 

(4.8) A,: G°^(X,E,) ^ C^{X,E,). 

It follows from ( |4.7| ) that Ay is symmetric and bounded from below. For / E Z put 



(4.9) c(a,) = ^ - 1. 
Define the operator A{a) acting on C^{X, E{a)) by 

(4.10) A{a):= A, + c{a) 

V 

Obviously, A{a) preserves the grading of E{a). 
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By ( ^.Tp the elliptic operator is symmetric and bounded from below. Therefore the 
heat operator e"*"^" is well defined and is a trace class operator. Given a G M, put 



(4.11) K{t; (j) = Y^ m^{a) Tr(e 



where m^{a) is defined by ( [4.4|) . Our next goal is to use the Selberg trace formula to 
express K{t, a) in terms of the length of the closed geodesies. 

Let A,y be the lift of Ajy to the universal covering X = G/K. It acts in the space of 
smooth sections of the homogeneous vector bundle E^, associated to z/. With respect to 
the isomorphism C°°{G/K,E^) = {C°°{G) ® K)^ we have 

A, = -R{n) ® Idy„ . 

Let e^*"^", t > 0, the heat semigroup generated by A^. This is a smoothing operator on 
L'^iG / Ey) = [E^iG) ® K/)^ which commutes with the action of G. Therefore it is of 
the form 

(e-*^^0) {g) = H^{g-'g')<P{g') dg', G {L'{G) ® V,)^ , g G G, 

where the kernel : G — )■ End(Vi/) is C°°, L^, and satisfies the covariance property 

(4.12) H^ik-'gk') = uik)-' o H^{g) o uik'), k,k' e K, g e G. 

Actually, a much stronger result holds. For g > let C^(G') be Harish-Cahndra's L^- 
Schwartz space. Then we have 



KxK 



(4.13) E iC''iG)®End{Vr)) 

for all g > 0. The proof is similar to the proof of Proposition 2.4 in |PM|| . By standard 
arguments it follows that the kernel of the heat operator e"*"^" is given by 

(4.14) H''{t;x,x') = J2Hti9-'l9'), 

where x,x' E X and x = TgK and x' = Tg'K. Therefore the trace of the heat operator 

Tr(e"*^'')= [ tiH''{t;x,x) dx, 
Jx 

where tr denotes the trace tr: End{Ei,^^) — )■ C for x G X. Let 

h'^{g) = tTH;:{g). 
Using i ^J% ) and (|]l|), it follows that 

(4.15) Tr(e-*^^)=/ Y.^t{9-'l9) dg. 

Jr\G ~^ 



7er 
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Let -Rr denote the right regular representation of G on LP'{T\G). Then (|4. 15|) can be 
written as 



(4.16) 

Let 

(4.17) 



Tr (e-*^-^) =Tri?r(/ij^) 



Then by (|]Tl|) and (^l6D we get 

K{t;a) = i:iRT{h1), t > 0. 

We can now apply the Selberg trace formula |[Wal|| . We use the notation introduced in 
section ^. Let n = ^(u), where 9 is the Cartan involution. For 7 G F \ {e} put 

D(7) = e^(^) det(Id- Ad(m^a^)n). 

Then the Selberg trace formula gives 

K{t-(y) = Yo\{X)h1{e) 

(4.18) + — V ^(^) 

27r ^ nr{i)D{^) 

Note that h1{e) can also be expressed in terms of characters. By ([4.13|) , each belongs to 
C'^{G) for all g > 0. Therefore is in C'^(G'). Hence we can apply the Plancherel formula 
for G (see |[Kn| , Theorem 11.2]). With respect to the normalizations of Haar measures used 
in |[Kn|| and the definition of the Plancherel polynom (|3.4|), we have 



„ ^ 1-7 •/ M 



(4.19) 



To continue we need to compute the characters Qn,xih1). First by ([4.17|) we have 
(4.20) QnAhl) = 5^m4(T)e„,A(/ir), 



which reduces the problem to the computation of QnAK)- For any unitary representation 
TT of G on a Hilbert space set 



n{g)®H^{g)dg. 



G 



This defines a bounded operator on "Htt ® V^. As in [ [BM| , pp. 160-161] it follows from 
( [4.12| ) that relative to the splitting 



7i'{H^) has the form 



<H-) 
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with T^{H^) acting on ("Htt ® K-) • Then it follows as in pM| , Corollary 2.2] that 
(4.21) 7r(/fn = e*"(^) Id, 

where Id is the identity on ("Htt ® K)^. Let {^njneN and {ej}^^ be orthonormal bases of 
"Htt and Vjy, respectively. Then we have 

oo m 

Tr7r(i7n = E Y.^<H^t){in ® e,), (^ ® e,)) 

n=l j=l 
oo m „ 

= X15Z / (^i9)U,U){Ht{9)ej,ej)dg 



(4.22) 



n=l i=l 



n=l •^'^ 



= Tr7r(/.r). 

Together with (|4.21|) we get 

(4.23) Tr 7r(/in = e*"^^) dim {U^ ® V^f . 

Now we consider a unitary principal series representation tt^^a Let [i/|m : cr^] denote the 
multiphcity of cr„ G M in z/|m- It equals or 1. For any representation vr of G denote by 
TT^ the contragredient representation of tt. By Frobenius reciprocity |[Kn] , p. 208], we have 

dim('H„,A ® K)^ = k^Ak^ '^l = [t^-u-xIk- H = [^^Im: ct-u] = W\m- (^n\- 



Combined with ( [4.23| ), we obtain 

e„,A(/ir)=e*"-^(^)[z/|M:fxJ. 
Using (g), O and (U), we get 

(4.24) ^n,xm = e*"-^(^) Yl : ^n] = e*"-^(^)[a + : a„]. 

The Casimir eigenvalue TTn,\{^) is given by ( ^.12| ). Using the definition of c{a) by ( [4. 91 ) it 
can be written as 

(4.25) 7r„,A(^^) = -A' + c(a„). 

Now we can put our computations together. Let k G Nq. If we insert (^1^ in ( ^ISD and 
( |4.19| ) and use (g]2|), we get 

ir(t;afc) = e*'=('^'=)(2vol(X) / e^'"'" P^,{tX) dX 



(4-26) 



£(7) e-^(^) 
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where 

det (Id — Ad(m^a^)n-) 
Using the definition of v4(cr) by (|4.1CI|) together with ( [4.11| ), we finally get 

Proposition 4.1. For every a E M we have 

(4.28) Tr. (e-^W) = 2vol(X) y^e^^ P.(.A) d\^Y^ 

5. The functional equation of the Selberg zeta function 



One of the main ingredients of the proof of Theorem is the functional equation ( |3.5| ) 
satisfied by the Selberg zeta function Z(s,cr). In particular, it is important to determine 
the sign in the exponential factor. We include a proof of the functional equation for the 
symmetrized Selberg zeta function which suffices for our purpose. 

Let a G M . Note that A{a^ is a second order elliptic differential operator on a compact 
manifold. Therefore it is essentially self-adjoint and the unique self-adjoint extension of 
yl(cr) has pure point spectrum consisting of a sequence of eigenvalues Ai < A2 < • ■ ■ — J- 00 
of finite multiplicities. It follows from Weyl's law that 

(5.1) 5^Ar2<oo. 

Ai>0 

Therefore the resolvent (A(cr) + s^)^^, Re(s^) 3> 0, is a Hilbert-Schmidt operator. Let 
Re(s^) ^ ,2 = 1, 2. By the resolvent equation we have 

(A(a) + s\)-' - (A(a) + sir' = {si - sl){A{a) + sly' o {A{a) + sl)-\ 

Thus the right hand side is a product of Hilbert-Schmidt operators and therefore, it is a 
trace class operator. Hence {A{a) + sf)^' — {A{a) + sl)'^' is a trace class operator. Now 
observe that 

POO 

{A{a) + s'')-' = / e-^^'e-*^^'^) dt. 
Jo 

Furthermore we have the heat expansion 



(5.2) Tr(e-^(-))~5^., 



as t +0. Let Re{s^), Re(sg) > 0. Then it follows from (U) that 

POO 

(5.3) Tr, {(Aia) + s'y' - (A(a) + si)-') = / (e"*^' - e'^^o) Tr, (e'^^^'^)) dt. 

Jo 
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Now we replace Tr^ (e ^^^'^^^ by the right hand side of ([4.28|) . First note that for Re(s) > 
we have 



(5.4) 



2s 



Furthermore by Cauchy's theorem we have 

e-'^'P^{iX) dX ] dt 



(5.5) 



TT 



(A2 + s2)(A2 + s2) 

So 



P{iX) dX 



-PJs 



For the last equality we used that -Po-(s) is an even polynomial. By ( |5.4| ) and (p.5|), we get 
(5.6) 



Tr, {{Aia) + s'y' - (A(a) + si)-') = 27rvol(X) 
2s f-^ nr 7 



PAs) PaiSo] 



-E 



So 
£(7) 



2^0(^^nr(7) 



-So£(7) 



By (3.3) we have 



Z'{s,a) ^ Z'{s,WAcr) 



Z{s, a) Z{s, Wa<^) ' 



which is the logarithmic derivative of the symmetrized Selberg zeta function S{s, a) defined 
by (|3.26|) . Thus we get 

'Pais) P^So) 



(5.7) 



Put 



Tr, {(Aia) + s^' ' (^M + ^o)"') =27rvol(X) 



+ 



S So 

1 S'{s,a) 1 S'{so,a 



2s S{s,a) 2sq S{so,a) 



E{s,a) = exp (^4nYo\{X) P^r) dr^ S{s,a). 
Then (|577|) can be rewritten as 



(5.8) 



Tr, {{A{a) + s'y' - {Aia) + si)-') 



1 E'is,a) 1 E'iso,a) 



2sE{s,a) 2soE{so,cr)' 

From this equality one can deduce the existence of the meromorphic extension of ^(s, a) and 
determine the location of the singularities, i.e., zeros and poles of S{s, a). Let Ai < A2 < ■ • • 
be the eigenvalues of A{a). For each Xj let S{Xj) be the eigenspace of A{a) with eigenvalue 



Xj. Put 



'ms{Xj,a) = dimgr £^(Aj). 
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If A,- < 0, we choose the square root \/A7 which has positive imaginary part. Put 



sf = ±i^/X'j, j e N. 

Proposition 5.1. The Selberg zeta function S{s,a), defined for R.e{s) > 2 by (|3.2| ), has 

a meromorphic extension to s G C. The set of singularities of S{s,a) equals {sf: j G N}. 
If \j 7^ 0, then the order of S{s,a) at both sf and sj is equal to ms{\j,a). The order of 
the singularity at s = is 2ms (0, o")- 



Proof. The left hand side of ( |5.8|) equals 



+ Xj Sq + Xj 



By ( ^.1|) the series converges absolutely and uniformly on compact subsets which shows 
that it is a meromorphic function of s G C and the only poles are simple and occur exactly 
at the points {sf : j G N}. Hence the logarithmic derivative of S(s,(t) is a meromorphic 
function with the same poles. Let Xj ^ 0. Then 

2s 1 1 

x + ^■ 



+ Xj S — Sj S — Sj 

It follows that sf are simple poles of a) ■ E{s, a)^^ with residue ms{Xj, a). Hence the 
order of S(s, a) at Sj equals ms{Xj, a). In the same way it follows that the order of S(s, a) 
at s = equals 2ms (0, cr)- D 

Now subtract from ( p. 81 ) the same equation for — s and multiply by 2s. Then we get 

^'is,a) ^ E'{-s,a) ^ 
S(s,a) S(-s,(t) 

which shows that the logarithmic derivative of • S(— equals zero. Therefore 
■ S(— s)^^ is constant. By Proposition |5.1| the order of S{s, a) at zero is even. Hence 

lim4M_ = l. 

This implies = S(— s). Since Pa{z) is even, we obtain the following functional equation 
for S{s, a): 

(5.9) S{s,a) = exp ^-Stt vol(X) P,(r) dr^ S{-s,a). 

Note that Z{s, (Xm) = Z(s, C-m)- Hence for s G M we have S{s, cr) = \Z{s, a) p. Then (|5] 
is reduced to 

(5.10) |Z(s,(t)| = exp f-47rvol(X) / P^{r) dr] \Z{-s,a)\, sG 
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6. The determinant formula 



By ||B0| , Theorem 3.19] the twisted Selberg zeta function can be expressed as a graded 



regularized determinant of A(a). We include a simple proof of this formula for our case. 

First we recall the notion of the graded regularized determinant of an elliptic self-adjoint 
operator. Let E = © E~ be a Z/2Z-graded Hermitian vector bundle over a compact 
Riemannian manifold. Let P: C°°{Y,E) — )■ C°°{Y,E) be an elliptic differential operator 
which is symmetric and bounded from below. Assume that P preserves the grading, i.e., 
assume that with respect to the decomposition 

C°°(F, E) = C+{Y, E+) © E~) 

P takes the form 

'P+ 
P 



P 



Then we define the graded determinant detgr(P) of P by 

. ^ , . ^ detfP+) 

(6.1) detgr(P) 



det(P-) 

Given a G M, let A{a) be the elliptic operator defined by (|4.1CI| ). It acts in a graded 
vector bundle. Hence the graded determinant detgr(s^ + A{a)) is defined. Let Po-(r) be 
the Plancherel polynomial ( p.4|) . By P0| , Theorem 3.19] the twisted symmetrized Selberg 
zeta function S{s, a) can be expressed by the graded determinant as follows. 

Proposition 6.1. We have 

(6.2) S{s] a) = detgr (s^ + A{a)) exp |^-47r vol(X) P^(r) dr^ , a ^ cxq, 
and 

(6.3) Z{s; do) = det (s^ - 1 + A) exp {(Gn)-^ vol(X)s^) , 

where A is the Laplace operator on C°°{X) and det is the usual regularized determinant. 

Proof. We give a simple proof of this formula. Let a G M . For Re(s^) ^ let 

POO 

(6.4) C{z, s)= e~*"' Tr, (e-*^(")) t'~^ dt. 

Jo 

The integral converges absolutely and uniformly on compact subsets of the half-plane 
Re(2;) > 3/2. It admits an extension to a meromorphic function of 2; G C which is 
differentiable in s. It is regular at 2; = and we have 

(6.5) C{z, s) = - log det(A(a) + s^) + 0{z). 
Furthermore for Re(z) > 3/2 we have 

(6.6) - ^£C(^, s) = e--^ Tr. (e^^^^)) ^ dt. 
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Thus by ( p.5[ ) we get 
1 d 



1 d 



s=so 



lim 



/■oo 

/ (e-*^' - e-*^o) Tr, (e-*^^'^)) dt. 
Jo 



(6.7) 



Assume that a ^ ao. Together with ( |5.3[ ) and (|5.7| ) we get 
for some 6 G C Integrating this equahty gives 



\s=so 



^ logdetgr {A{a) + s'^) = log a) + 47r vol(X)P„(s) + 6s 



(6.9) 



r b 

log S{s, a) = -47r vol(X) / P^(r) rfr + log detg^ {A{a) + s^) + -s'^ + c 

Jo 2 



for some c G C. In order to determine the constants b and c we take s G M and consider the 
asymptotic behavior of both sides of (|6.9|) as s — )■ oo. By (|3.3| ) it follows that log S{s, a) 
as s -> oo. Next consider the behavior of logdetgr {A{a) + s^) for s G M and s — )■ oo. By 
(WM we have 



(6.10) 



d 



logdetgr {A{a) + s^) = -— (z^z, s)) 



\z=0' 



Denote the first term on the right hand side of ( [4.28| ) by I{t, a) and the second by H{t, a). 
Let Re(^) > 3/2. Then by ( ^ ) and (|3) we get 

(6.11) ((z,s)= e-'''l{t,a)t'-^ dt+ e-''' H{t, a)t'-^ dt. 

Jo Jo 

It follows from the definition of H{t, a) that the integral e~*''^if(t, a)t^~^ dt is an entire 
function of 2; G C and for every compact subset w C C there exist C,c> such that 



(6.12) 



d r°° 

-- / e-'''H{t,a)t''^ dt 
dz Jo 



<C e 



z & uj, s > 0. 



To deal with the first integral on the right hand side of ( |6.11| ), we note that 



(6.13) e"*^' (^j e-'^-'X^^ dXj dt = T{j + l/2)r(-j - 1/2 + z)s 



2j-2z+l 



By ( p.4| ) the Plancherel polynomial Pa{z) is of the form Pa{z) = ai + a2Z^. Using the 
definition of /(t, cr) and ( |6.13| ), we get 

^ ^ _4^vol(X) (ais-|s3 



(6.14) 



dz 



e-'" I{t,a)t'-' dt 



z=0 



-47rvol(X) [ P^{r) dr. 
Jo 
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Together with (j^JUj ), (|01| ), and (1021) we obtain 



logdetgr (A(or) + s^) = 47rvol(X) / P^(r) dr + 0{ 

Jo 



for s G M, s — i- oo. This imphes that the constants b and c in (|6.9| ) are zero. Exponentiating 
( p.9|) , we get (|6.2| ). The proof of ( |6.3|) is similar. □ 



Remark. From the statement of Theorem 3.19 in | |BU| | it is not apparent that the deter- 
minant is the graded determinant. However, it is the general understanding in [ [B0|| that 
the trace of a trace class operator on a graded bundle is the super trace corresponding to 
the grading (see |PC)| , p. 29]). Consequently regularized determinants of elliptic operators 
on graded bundles are always understood in P0|| as graded determinants. 

Now let T be an irreducible, finite-dimensional representation of G with highest weight 
At- = {m,n). For w G Wq let ar^w ^ M and A^^^ be defined by ( |3.21| ). Let 

(6.15) A{w)= A, + T{n). 

V 

This is an elliptic operator acting on C°°(X, E[ar^w)\ Using ( |3.31|) , an explicite computa- 
tion shows that for all w G Wq we have 

(6.16) + c(a,,^) = ^ (m(m + 2) + n{n + 2)) = r(fi). 

Using ( |6.16| ), and (|6.15|) , it follows that 

(6.17) A{ar,^) + \\^^ = A{w) 

as operators on C°°{X, E{ar^w))- Then it follows from ( |6.2D that 

-47rvol(X) J Pa^^{r)drj, 

if (Tr^w 7^ o"o. If <7r,w = ctq, wc usc ( |6.3|) , which leads to a similar formula 
Proposition 6.2. Let tq ^ r. There is a constant c = c(r) such that 

(6.18) /?,(s)/?,,(s) = e"°'W^ J] detgr(s2 - 2A,,^s + A(«;))(-^)'''^'^\ 

Proof. By assumption we have r = r^.n with m ^ n. It follows from ( |3.31| ) that ar,w ^ o"o 
for all w eWc- Put 

i^.T.Jr) dr. 

Then it follows from {^^^ and (§3) that 

(6.20) Rr{s)Rr,{s) = e-4-™H^)^W JJ detg^ {s^ - 2sA,,^ + A(t/7))^"'^'*""^' . 

w^Wa 
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Using ( p.4p and ( |3.31| ), an explicite computation gives 

F{s) = -n-^{m + l)(n + l)s. 

Now we consider the case Tg = r. Then r = Tm,m for some m G Nq. 
Proposition 6.3. Let m G No- There exists a constant c = c(m) such that 

,cvoi(x)s . det ((g + m + 1)^ - 1 + A) det ((g - m - 1)^ - 1 + A) 



□ 



(6.21) Rr^,Js)=e'^ 
Proof. Put 



detgr + v4(cr2m+2)) 



i^m(s) = TT {{s + m + if + {s 

DTT 



^0 

Using ( CT ), (iD and (U), it follows that 

(^) = eVoi(X)F™(s) . det ((g + m + 1)2 - 1 + A) det ((g - m - 1)^ - 1 + A) 

detgr + A(cr2m+2)) 

Using (|]4D, it follows that F^{s) = 2Ti-\m + 1)^5. 



□ 



7. Proof of Theorem 1.5 



Since 



has not been published yet, we include a proof of Theorem |1.5|. Let r: G — ?► 



GL{Vr) be an irreducible finite-dimensional representation with associated flat bundle E^- 
equipped with an admissible metric. Let Ap(r) be the Laplacian on iJ^-valued p-forms. 
Let 



(7.1) 



and 



K{t,T) :-- 



3 



-tAp(T) 



) 



g(r) = ^{-iypdimkeTHP{X,Er). 
p=i 

Then by definition of the analytic torsion we have 
(7.2) log Tx(r) = 14: (t:^ r iK{t,T)-q{r))t^'' dt 



2ds Wis 



s=0 



where the right hand side is defined near s = by analytic continuation of the Mellin 
transform. The first step of the proof is to apply the trace formula to express K{t, r) in 
terms of the length of closed geodesies. This is the basis for the relation between analytic 
torsion and the twisted Ruelle zeta function. 

Let p be the orthogonal complement of t in g with respect to the Killing form. Let 
xq = eK. Recall that there is a canonical isomorphism Txq{G/K) = p. Let Rr denote the 
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right regular representation of G on L'^{V\G) (resp. C°°(r\G)). Using (p.l5| ), we get a 
canonical isomorphism 

(7.3) AP(X, Er) = (C°°(r\G') ® A^p* ® K)^ , 

where K acts by G -ft' i— ?■ -Rr(^) ® A''' Ad* (A;) (g) r{k). There is a similar isomorphism for 
the space L^A^(X, Er) of L^-sections of ApT*X ® -Er- With respect to the isomorphism 
([7.3|), we have the following generalization of Kuga's lemma 



(7.4) Ap(r) = --Rr(fi) » W +r(r]) Id, 

(see | |1V1M| , (6.9)]), where Q is the Casimir element and t{Q) is the Casimir eigenvalue of r. 

Let Ap(r) be the lift of Ap(r) to the universal covering X = G/K. Let e~^'^p^'^\ t > 0, 
be the corresponding heat semigroup. This is a smoothing operator on 

L^Af{X; Er) = {L\G) ® A^p* ® K)^, 

which commutes with the action of G. Therefore, it is of the form 

(e-*^^M0) (g) = I Hl'^{g-'g')ct>{g') dg', G (L^G) ® A^p* ® K)^, ^7 € G, 

where the kernel H^'^ : G End(APp*®K-) belongs to G°°r\L'^ and satisfies the covariance 
property 

(7.5) Hl'^ik-'gk') = u,ir){k)''Hl'^{g)u,iT)ik'), 
with respect to the representation 

(7.6) Upir) := A^Ad^^r: K GL(APp* ® K). 
Moreover, for all g > we have 

(7.7) H^'P e (C'?(G) ® End(APp* ® K))^""^, 

where C'^{G) denotes Harish-Cahndra's -L^-Schwartz space. The proof is similar to the 
proof of Proposition 2.4 in [ [BM| | . Let 

hl%g) = tTHl^^{g). 

Repeating the arguments which we used to prove ( [4.16|) , we get 

(7.8) Tr (e-*^-M) =Tr-R^(/i[''')• 
Put 

3 

(7.9) ki = Y^{-irphi^^. 

p=i 

By (|7. 1|) we have 

K{t,T) = TiRrikl). 
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We can now apply the Selberg trace formula ||Wal|| . Let the notation be as in ( [4.18p . Then 

we get 

ir(t,r)=Vol(X)fc[(e) 
(7.10) , 1 ^(7) 



27r f-r! nr(7)^(7) 

where the notation is the same as in ( [4.18 ). The characters Qn,x{kJ) can be computed in 
the same was as in section ^. Let vr be a unitary representation of G on a Hilbert space 
"Htt- Set 



T,p\ 



nig)®Hrig)dg. 



G 



This defines a bounded operator on ® A^p* Vr. As in [[BM| , pp. 160-161] it follows 
from ( |7.5| ) that relative to the splitting 



'k{Hl'^) has the form 



7r(i7-^) 




with Ti{Hl'^) acting on {H^ ® A^p* ® K) • Using (jT^) it follows as in [|BM|, Corollary 2.2] 
that 

vr(i7[''') = e*("(^)-^(^)) Id 
on {H^ ® APp* ® Vr)^. As in d^H) we get 

(7.11) tr7r(i7;'^)=tr7r(/.n. 

Now let vr be a unitary principal series representation 7r„ a- Using ( 7.11 ) and ( 2.12|) we get 

(7.12) QnAK") = e-*(^'+i-"'/4+-(^)) dim (H„ ® A^p* ® K)"" • 

Denote by C„ the M-module defined by cr„. By Frobenius reciprocity ||Kn| , p. 208] we have 



dim {H^^x ® A^p* ® Vrf = dim (C„ A^p* ® K 



and by ( [7. 9]) we get 



e„,A(A;[) = e-*(^'+i-"'/4+^(^)) ^(-l)Pp dim (C„ ® A^p* ® K)"^ • 

p=i 

Choose an orthonormal basis of p as in ||Mi| , p. 9]. Using this basis it follows that as 
M-modules, p and a © n are equivalent. Thus we get 

33 2 
(7.13) J](-l)PpAfp* = J](-l)^p (A^n* + A^-^n*) = J](-l)P+^Afn*. 
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Therefore, the Fourier transformation of kl is given by 



(7.14) e^AK) 



-t{X'^+l-n' 



74+r(Q)) J2^_iY+i dim(C„ ® A^'n* ® Vr 



M 



p=0 



This formula can be simphfied using the real version of Konstant's Bott-Borel-Weil theorem 
We apply Lemma to determine the G Z for which dim(C„ ® A^n* K-)*^ 7^ 0. 
We decompose the characters on the right hand side of ( |3.18| ) according to ( p.21| ). Let 
a^^yj e M and A^^^ G |Z be defined by (|3.21|) . Using ( p. 181) and ( |6.16|) , we get 



(7.15) 



J] (-l)^(-)+V.,^(m,)e 

w^Wq 



-^(7)V(4t) 



(47rt)i/2 • 

Next we consider the contribution of the identity to (|7.10|) . By ( [r.7|) , is in C^iG) for 
all g > 0. Therefore we can apply the Plancherel formula for G (see [[Kn| , Theorem 11.2]). 
With respect to the normalizations of Haar measures used in [ [Kn|| , we have 



kl{e) = Y. [ e„,,(/c[)P.„(zA)rfA, 



where Pa„{z) is the Plancherel polynom (^.4]). Repeating the arguments that led to (|7.15|) , 

we get 



/ e-*^'P._(a)rfA. 
Jr 



(7.16) 

Combined with ( |7.10|) and (|7.15|) , we obtain 

Kit,r)= 5^ (-l)^(-)+ie-*^-(vol(X) [ e-*^>.,, JzA) rfA 



(7.17) 



+ 2^ r^^(7;^r. 



-^(7)V{4i) 



where L(7, cr) is defined by 

(T.18) L(7,a) 



a[m^)e 



(47rt)i/2 



det (Id — Ad(m^a^)n-) 



Unfortunately, the constants At- „, appearing in the exponential factors prevent us from 
applying the Mellin transform to this formula directly. This problem occurred already in 
^ . To overcome this problem we use the auxiliary operators introduced in section 0. 
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Using ( [4 .261 ) and ( |6.16| ), it follows that for w G Wg we have 



e^(^)*ir(t,a,,») = e-*^-'f2vol(X) / e-*^>.,, J^A) t/A 
(7-19) ^/ N e-^W'/{4i)\ 



[7]^ 

Next observe that by ( p.31| ) there exists a decomposition 

with = 2, z = 1, 2, and a bijection j : PVo — ^ Wi such that for w G VFq we have 
Hence by ( [7.17P and ( |7.19| ) we get 



(7.20) r) = ^ 5^ (-l)^(-)+ie-(^)*ir(t; a.,^). 

w^Wq 

This equality can be expressed in a slightly different way as follows. Denote by Tr^ the 
supertrace with respect to the grading of E{ar,w)- Using the definition of K{t,ar,w) by 
( [4.11| ) and the definition of A{w) by ( p.l5| ), we get 



(7.21) K{t, r) = ^Yl (-1)'^"^+' Tr^ (e-*^^"')) . 

weWo 

To continue we need to determine the location of the spectrum of the operators A(cr). 
Lemma 7.1. For a E M we have A{(t) > —1. Moreover, if k ^ {0, ±2}, then A{ak) > —1. 

Proof. Let G denote the unitary dual of G. Let 

(7.22) L\r\G) = 0^^^mr(vr)H. 

be the spectral decomposition of the right regular representation of G on L^(r\G). Let 
(z/, V,y) be an irreducible unitary representation of K. Then L^(X, E,y) = (L^(r\G) ® V,^)^ . 
Using (Q), we get 



(7.23) (^'(r\G) ® K)"^ = ® ~mr(7r) (H. ® V,f . 

This decomposition corresponds to the spectral resolution of as follows. Assume that 
mr(vr) dim('H,r ® Vi,)^ 7^ 0. Then mr(7r)('H,r ® K)^ is an eigenspace of A^, with eigenvalue 
— 7r(f2). Note that G is the union of the trivial representation, the unitary principal series 
vTyfc A with G Z and A G M, and the complementary series vr^ with < x < 1 |p<.S| , Propo- 
sition 49], ||Kn| , Theorem 16.2] (where for the latter reference the different parametrization 
of the induced representations has to be taken into account). 
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First consider the principal series 7Tn,\- By Frobenius reciprocity |[Kn| , p. 208] we have 
for leNo 



(7.24) dim('H^^^^ (g) = ['^k,\\K ■ J^i] = [j^iIm ■ = hU/ ■ ^k]- 

By (|]Tp it follows that (H^,^ 7^ implies / > k. Moreover, by (g^) it follows that 

^uiio'm) 7^ implies m >l. Thus if my{am) 7^ and {H-,,^^ ® V^)^ ^ 0, then we have 
m > k. Hence ii u G K and a G M are such that m^{a) ^ and (^-Ki. ^ ® V",y)^ 7^ 0, then 
it follows from (^J^) and that 

(7.25) - 7ik,x{n) + c(or) > 0. 

Next consider the complementary series. By ( [4. 91 ) we have c{a) > — 1 for all a G M. Since 
< X < 1, it follows from (PTT^ ) that 

(7.26) _7r^(f]) + c(a)>-l. 



for all 0" G M. Finally, the trivial representation of G occurs in (|7.23|) only if u is the 
trivial representation uq. Moreover, by ( [4.2|) we have my^^^ai) 7^ 0, only if Z = or Z = 2. 
Thus by ( [7.25|) , (|7.26|) , and the definition of A{a) by ( [4.10|) , the statement of the Lemma 
follows. □ 



We apply this lemma to study the kernel of the operator A(w), w G Wg-, which is defined 
by (|15D. 

Lemma 7.2. Let r he an irreducible, finite- dimensional representation of G. Assume that 
Te ^ T. Then ker A(w) = {0} for all w G Wg- 



Proof. Let r = r^^^ with m ^ n. We use (|6.17|) to express A(w) in terms of Ai^ar^)- By 
(^) we have A^'^ G \ {0} for all w G Wg- If |A^,^| > 1, it follows from (§3^) that 
A{w) > 0. It remains to consider the cases Xr,w = ±1 and Ar,to = ±1/2. In the first 
case we have \m — n\ = 2. Then it follows from ( 3.31 ) that a^^w = <^2i with |Z| > 2. By 
Lemma [7.1| we get A{w) > 0. In the second case we have \m — n\ = 1. By ( |3.31D it follows 
that ar,w = (^21+1 for some / G Z. Let u E K such that m,y{o'2i+i) 7^ 0. By ( [4.2P there 
exists p G No such that z/ = i^2p+i- Since vr^. is induced from the trivial representation and 
['^2p+i|m : ctq] = 0, Frobenius reciprocity \^n[ p. 208] implies 



(7.27) 



dim('H^c (g) V,, 



VxW ■ l^2p+l] = [i^2p+i\m ■ CTq] 



0. 



Thus in this case the complementary series does not occur in ( |7.23| ). Also the trivial 
representation does not occur. By (|7.25| ) it follows that A{(7r,w) > 0. Using ( p. 17]) we get 
A(w) > 0. ' □ 



Now we can turn to the proof of Theorem p..5| . First assume that t ^ Tg. Then it 
follows from PW] , Chapt. VII, Theorem 6.7] that H*{X,Er) = 0. Hence Ap(r) > for 
all p, < p < 3. By Lemma 772 we also have A{w) > 0, w G Wg- Hence K{t,T) and 
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Tr(e *'^("'))^ uj G Wq-, are exponentially decreasing as t — t- oo. Therefore we can take the 
Mellin transform of both sides of ( [7.21| ) and we get 

which holds for Re(s) > 3/2. After analytic continuation we compare the derivatives at 
s = of both sides. Using (|T^) we get 



(7.28) 



Tx(rr = ndet(A,(r)r 



n detgr(A(t.))( 



Now we use the determinant formula (|6.2|) to relate the right hand side to the value at zero 
of the Ruelle zeta function. Since A(w) > 0, it follows that detgr(s^ — 2sXr,w + A(w)) is 
regular at s = and its value at s = is equal to detgr {^{w)) ^ 0. Hence 

lim JJ detgr (s^ -2sA^,^ + A(w))^"^^''™'''' = JJ detg, (A(u;))^ 



(7.29) 



w£Wg 



■w£Wg 



By ( |S.20| ) it follows that Rr{s)Rrg{s) is regular at zero. Now observe that r = tq. Further- 
more by ( |3.13D we have 

Rr{s) = Rr,is). 

This implies that Rt{s) is regular at s = and 

(7.30) l^r(0)|'= n detg,(A(«;))(-')'*""^\ 

weWc 

Combining (|7.28|) and ( [7. 301) , the first statement of Theorem p..5| follows. 

Next assume that Tg = r. Then there exists m G Nq such that r = r^,™- We use ( |6.21| ) 
to determine the order of Rt{s) at s = 0. For m G No let 

(7.31) /im = dimgrker(A(a2m+2)), 

where dimgj. denotes the graded dimension of a graded vector space, i.e., ifV = © V~ is 
a graded finite-dimensional vector space, then dimgj. V = dimK"^ — dim\^~. Assume that 
m > 1. Then det {{s ± (m + 1))^ — 1 + A) is regular and nonzero at s = 0. Furthermore, 
detgr (s^ + A{a2m+2)) has order 2hm at s = and 

(7.32) lim s-^'^™ detgr {s^ + ^(^^2^+2)) = detgr iA{a2m+2)) ■ 
By (|6.21|) , it follows that Rt{s) has order —2hm at s = and we have 



det ((m+ 1)^ 



A)' 



(7.33) lim s"^-Rr,^,Js) 

s^O detgr (^(0-2»n+2)) 

On the other hand, using (|3.32 ), it follows from ( [7.21|) that 

(7.34) K{t, t) = Tts (e-*^('^2™+2)) _ 2e-*((™+i)'-i) Tr (e^*^) . 
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Taking the limit t — t- oo of both sides of this equahty, we get 

3 

(7.35) hm = 5^(-lfpdim(ker Ap(r)) . 

p=i 



Moreover (|7.34| ) also implies 



Combining this equality with ( [7 .331) and ( [r.35| ), we obtain the second statement of Theorem 



8. Proof of Theorem 1.1 



We are now ready to prove our main result. We consider the representation r^. Using 
(PD, it follows from (^)that 



(8.36) 



Rr^{s) = l[R{s-{m/2-k),a. 



m—2k) 



k=0 



We distinguish the cases where m is odd and even. Let m > 3. Then we get 



(8.37) 



fc=0 



k=3 



= Rr^{s)Y\_R{s - k,a2k)R{s + k,a-2k)- 

fc=3 

Similarly, for m > 2 we get 

m 

(8.38) Rr,^^, {s) = Rr,{s) n ^(^ - ^ - 1/2, (r2k+i)R{s + k + 1/2, a_(2,+i)). 

k=2 

Now recall that by Proposition p.2| , 1), each R{s,ai), / G Z, is regular in the half-plane 
Re(s) > 2 and does not vanish in this half-plane. By the functional equation ( p. 11 ) the 
same holds in the half-plane Re(s) < 2. Therefore the products on the right hand side of 
( ^.3?! ) and (|8.38|) are regular at s = 0. Furthermore it follows from ( p.6|) that 



(8.39) \Ris,ai)\ = \Ris,a. 

Using ( ^.37] ), ( p. 381) and Theorem ( |1.5|) we get 



(8.40) 



Tx{T2mf = Tx{ufl[\R{k,a2k)\ ■ \R{-k, o-2k)\, m>3. 



k=3 
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and 

m 

(8.41) Tx{T2rn+i? = Tx{r^f n + 1/2, fX2fc+i)| ■ \R{-k - 1/2, a2fe+i)|, m > 2. 

fe=2 

By the functional equation ( p.ll| ) and ( |8.39| ) we get 

4 



\R{-k,(j2k)\=e^-9 ( --vol(r\e=^)A; ) \R{k,a2k 



TT 



Together with ( |8.40| ) this leads to 



(8.42) Tx{r2m)=Tx{u)X{e^v[ — ^o\{T\n^)k\ \R{k,a2k] 



k=3 

Similarly 



(8.43) TxiT2^+i)=Tx{n)l[expl vol(r\e=')(A: + 1/2) j \R{k + 1/2, a2k+i)\. 

k=2 ^ ^ ^ 

To continue we need the following estimation. 
Lemma 8.1. There exists C > such that for all m & N , m > 3 , we have 

m m 

Y,\^og\R{k,a2k)\ \ <C, ^|log|i?(A; + l/2,a2fc+i)l| < C. 

k=3 k=2 

Proof. We consider the first case. Since |(T2fe(m^)| = 1, we have 

1 _ e-'^^) < |l - a2fc(m^)e-'^^(^)| < 1 + e-'^^^^l 

Let k > 3. Using that the infinite product ( p.6| ) is absolutely convergent for Re(s) > 2, we 
get 

^log(l-e-'=^(^)) <log|i?(A;,a2,)| < 5^1og(l + e-^-^W), 



prime prime 



which implies 



oo ^ 

(8.44) |log|i?(A;,a2fc)|| <Y.Y.- 



g-nM(7) 



n 

[-y]jte n=l 
prime 
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Let 6 = mf{£(7) : 7 G T \ {e}}. Put Ci = (1 - e 



-<5\-l 



Using (p.44) we get 



-£(-f)nk 



fc=3 



n 

n=l [7]^e k=3 
prime 



^1 /I _ g-n(m+l)£(7) 

^ ^ I 1 - e-'^^(T) 

n=l [7]^e ^ 



-(1 + 



prime 

n=l [7]^e 
prime 



-{m+l)n£{7) 



< 



[7]^e n=l 
prime 



1 _ e-n^(7) 
3n^(7) 



= Cilogi?(3,ao)-^ = a 

The other case is similar. 
Taking the logarithm of both sides of ( ^.42 ) and ( |8.43| ), respectively, we obtain 

m ^ 

logTx{T2m) = logTxiu) + V log \R2k{k)\ - - vol(r\M3) (m(m + 1) - 6) . 

and 



fc=3 



log Tx (T2m+1 ) = logTx(r3) + ^log 

Applying Lemma we get 

- log Tx(r^) 



□ 



- vol(r\e^)(m(m + 2) - 3). 

TT 



vol(r\e3)m2 + 0{m 



as m — !■ 00. This completes the proof of Theorem |L1| . 
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